An examination of the reliability of prestigious scholarly journals: evidence and implications for decision-makers

In universities all over the world, hiring and promotion committees regularly hear the argument: “this is important work because it is about to appear in prestigious journal X”. Moreover, those who allocate levels of research funding, such as in the multi-billion pound Research Assessment Exercise in UK universities, often come under pressure to assess research quality in a mechanical way by using journal prestige ratings. This paper’s results suggest that such tendencies are dangerous. It uses total citations over a quarter of a century as the criterion. The paper finds that it is far better to publish the best article in an issue of a medium-quality journal like the Oxford Bulletin of Economics and Statistics than to publish the worst article (or often the worst 4 articles) in an issue of a top journal like the American Economic Review. Implications are discussed

An index to quantify an individual's scientific research output that takes into account the effect of multiple coauthorship

I propose the index $\hbar$ ("hbar"), defined as the number of papers of an individual that have citation count larger than or equal to the $\hbar$ of all coauthors of each paper, as a useful index to characterize the scientific output of a researcher that takes into account the effect of multiple coauthorship. The bar is higher for $\hbar$.Comment: A few minor changes from v1. To be published in Scientometric

Runaway Events Dominate the Heavy Tail of Citation Distributions

Statistical distributions with heavy tails are ubiquitous in natural and social phenomena. Since the entries in heavy tail have disproportional significance, the knowledge of its exact shape is very important. Citations of scientific papers form one of the best-known heavy tail distributions. Even in this case there is a considerable debate whether citation distribution follows the log-normal or power-law fit. The goal of our study is to solve this debate by measuring citation distribution for a very large and homogeneous data. We measured citation distribution for 418,438 Physics papers published in 1980-1989 and cited by 2008. While the log-normal fit deviates too strong from the data, the discrete power-law function with the exponent $\gamma=3.15$ does better and fits 99.955% of the data. However, the extreme tail of the distribution deviates upward even from the power-law fit and exhibits a dramatic "runaway" behavior. The onset of the runaway regime is revealed macroscopically as the paper garners 1000-1500 citations, however the microscopic measurements of autocorrelation in citation rates are able to predict this behavior in advance.Comment: 6 pages, 5 Figure

Towards a new crown indicator: an empirical analysis

We present an empirical comparison between two normalization mechanisms for citation-based indicators of research performance. These mechanisms aim to normalize citation counts for the field and the year in which a publication was published. One mechanism is applied in the current so-called crown indicator of our institute. The other mechanism is applied in the new crown indicator that our institute is currently exploring. We find that at high aggregation levels, such as at the level of large research institutions or at the level of countries, the differences between the two mechanisms are very small. At lower aggregation levels, such as at the level of research groups or at the level of journals, the differences between the two mechanisms are somewhat larger. We pay special attention to the way in which recent publications are handled. These publications typically have very low citation counts and should therefore be handled with special care

Peer review quality and transparency of the peer-review process in open access and subscription journals

BACKGROUND:Recent controversies highlighting substandard peer review in Open Access (OA) and traditional (subscription) journals have increased the need for authors, funders, publishers, and institutions to assure quality of peer-review in academic journals. I propose that transparency of the peer-review process may be seen as an indicator of the quality of peer-review, and develop and validate a tool enabling different stakeholders to assess transparency of the peer-review process. METHODS AND FINDINGS:Based on editorial guidelines and best practices, I developed a 14-item tool to rate transparency of the peer-review process on the basis of journals' websites. In Study 1, a random sample of 231 authors of papers in 92 subscription journals in different fields rated transparency of the journals that published their work. Authors' ratings of the transparency were positively associated with quality of the peer-review process but unrelated to journal's impact factors. In Study 2, 20 experts on OA publishing assessed the transparency of established (non-OA) journals, OA journals categorized as being published by potential predatory publishers, and journals from the Directory of Open Access Journals (DOAJ). Results show high reliability across items (α = .91) and sufficient reliability across raters. Ratings differentiated the three types of journals well. In Study 3, academic librarians rated a random sample of 140 DOAJ journals and another 54 journals that had received a hoax paper written by Bohannon to test peer-review quality. Journals with higher transparency ratings were less likely to accept the flawed paper and showed higher impact as measured by the h5 index from Google Scholar. CONCLUSIONS:The tool to assess transparency of the peer-review process at academic journals shows promising reliability and validity. The transparency of the peer-review process can be seen as an indicator of peer-review quality allowing the tool to be used to predict academic quality in new journals

Truncation of power law behavior in "scale-free" network models due to information filtering

We formulate a general model for the growth of scale-free networks under filtering information conditions--that is, when the nodes can process information about only a subset of the existing nodes in the network. We find that the distribution of the number of incoming links to a node follows a universal scaling form, i.e., that it decays as a power law with an exponential truncation controlled not only by the system size but also by a feature not previously considered, the subset of the network ``accessible'' to the node. We test our model with empirical data for the World Wide Web and find agreement.Comment: LaTeX2e and RevTeX4, 4 pages, 4 figures. Accepted for publication in Physical Review Letter

Identifying Overlapping and Hierarchical Thematic Structures in Networks of Scholarly Papers: A Comparison of Three Approaches

We implemented three recently proposed approaches to the identification of overlapping and hierarchical substructures in graphs and applied the corresponding algorithms to a network of 492 information-science papers coupled via their cited sources. The thematic substructures obtained and overlaps produced by the three hierarchical cluster algorithms were compared to a content-based categorisation, which we based on the interpretation of titles and keywords. We defined sets of papers dealing with three topics located on different levels of aggregation: h-index, webometrics, and bibliometrics. We identified these topics with branches in the dendrograms produced by the three cluster algorithms and compared the overlapping topics they detected with one another and with the three pre-defined paper sets. We discuss the advantages and drawbacks of applying the three approaches to paper networks in research fields.Comment: 18 pages, 9 figure

Electron impact excitation cross sections for allowed transitions in atoms

We present a semiempirical Gaunt factor for widely used Van Regemorter formula [Astrophys. J. 136, 906 (1962)] for the case of allowed transitions in atoms with the LS coupling scheme. Cross sections calculated using this Gaunt factor agree with measured cross sections to within the experimental error.Comment: RevTeX, 3 pages, 10 PS figures, 2 PS tables, submitted to Phys. Rev.